
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <link href='/css/styles.css' rel='stylesheet' type='text/css' />
    <link href='/images/favicon.png' rel='shortcut icon' />
    <script src='/js/jquery.min.1.4.js'></script>
    <script src='/js/app.js'></script>
    <meta content='width=device-width, minimum-scale=1.0, maximum-scale=1.0' name='viewport' />
    <title>Redis中文资料站</title>
	<meta http-equiv="description" content="redis中文资料站，下载安装redis，查找redis常用命令（commands），选择适合的redis客户端方式，配置redis主从（master-slave），阅读redis官方文档，社区里了解更多redis信息，提交redis的bug。" />
  </head>
  <body class=''>
    <script src='/js/head.js'></script>
    <div class='text'>
      <article id='topic'>
        <h1>Redis latency problems troubleshooting</h1>
        
        <p>This document will help you understand what the problem could be if you
        are experiencing latency problems with Redis.</p>
        
        <p>In this context <em>latency</em> is the maximum delay between the time a client
        issues a command and the time the reply to the command is received by the
        client. Usually Redis processing time is extremely low, in the sub microsecond
        range, but there are certain conditions leading to higher latency figures.</p>
        
        <h2>Measuring latency</h2>
        
        <p>If you are experiencing latency problems, probably you know how to measure
        it in the context of your application, or maybe your latency problem is very
        evident even macroscopically. However redis-cli can be used to measure the
        latency of a Redis server in milliseconds, just try:</p>
        
        <pre><code>redis-cli --latency -h `host` -p `port`&#x000A;</code></pre>
        
        <h2>Latency induced by network and communication</h2>
        
        <p>Clients connect to Redis using a TCP/IP connection or a Unix domain connection.
        The typical latency of a 1 GBits/s network is about 200 us, while the latency
        with a Unix domain socket can be as low as 30 us. It actually depends on your
        network and system hardware. On top of the communication itself, the system
        adds some more latency (due to thread scheduling, CPU caches, NUMA placement,
        etc ...). System induced latencies are significantly higher on a virtualized
        environment than on a physical machine.</p>
        
        <p>The consequence is even if Redis processes most commands in sub microsecond
        range, a client performing many roundtrips to the server will have to pay
        for these network and system related latencies.</p>
        
        <p>An efficient client will therefore try to limit the number of roundtrips by
        pipelining several commands together. This is fully supported by the servers
        and most clients. Aggregated commands like MSET/MGET can be also used for
        that purpose. Starting with Redis 2.4, a number of commands also support
        variadic parameters for all data types.</p>
        
        <p>Here are some guidelines:</p>
        
        <ul>
        <li>If you can afford it, prefer a physical machine over a VM to host the server.</li>
        <li>Do not systematically connect/disconnect to the server (especially true
        for web based applications). Keep your connections as long lived as possible.</li>
        <li>If your client is on the same host than the server, use Unix domain sockets.</li>
        <li>Prefer to use aggregated commands (MSET/MGET), or commands with variadic
        parameters (if possible) over pipelining.</li>
        <li>Prefer to use pipelining (if possible) over sequence of roundtrips.</li>
        <li>Future version of Redis will support Lua server-side scripting
        (experimental branches are already available) to cover cases that are not
        suitable for raw pipelining (for instance when the result of a command is
        an input for the following commands).</li>
        </ul>
        
        <p>On Linux, some people can achieve better latencies by playing with process
        placement (taskset), cgroups, real-time priorities (chrt), NUMA
        configuration (numactl), or by using a low-latency kernel. Please note
        vanilla Redis is not really suitable to be bound on a <strong>single</strong> CPU core.
        Redis can fork background tasks that can be extremely CPU consuming
        like bgsave or AOF rewrite. These tasks must <strong>never</strong> run on the same core
        as the main event loop.</p>
        
        <p>In most situations, these kind of system level optimizations are not needed.
        Only do them if you require them, and if you are familiar with them.</p>
        
        <h2>Single threaded nature of Redis</h2>
        
        <p>Redis uses a <em>mostly</em> single threaded design. This means that a single process
        serves all the client requests, using a technique called <strong>multiplexing</strong>.
        This means that Redis can serve a single request in every given moment, so
        all the requests are served sequentially. This is very similar to how Node.js
        works as well. However, both products are often not perceived as being slow.
        This is caused in part by the small about of time to complete a single request,
        but primarily because these products are designed to not block on system calls,
        such as reading data from or writing data to a socket.</p>
        
        <p>I said that Redis is <em>mostly</em> single threaded since actually from Redis 2.4
        we use threads in Redis in order to perform some slow I/O operations in the
        background, mainly related to disk I/O, but this does not change the fact
        that Redis serves all the requests using a single thread.</p>
        
        <h2>Latency generated by slow commands</h2>
        
        <p>A consequence of being single thread is that when a request is slow to serve
        all the other clients will wait for this request to be served. When executing
        normal commands, like <strong>GET</strong> or <strong>SET</strong> or <strong>LPUSH</strong> this is not a problem
        at all since this commands are executed in constant (and very small) time.
        However there are commands operating on many elements, like <strong>SORT</strong>, <strong>LREM</strong>,
        <strong>SUNION</strong> and others. For instance taking the intersection of two big sets
        can take a considerable amount of time.</p>
        
        <p>The algorithmic complexity of all commands is documented. A good practice
        is to systematically check it when using commands you are not familiar with.</p>
        
        <p>If you have latency concerns you should either not use slow commands against
        values composed of many elements, or you should run a replica using Redis
        replication where to run all your slow queries.</p>
        
        <p>It is possible to monitor slow commands using the Redis
        <a href="/commands/slowlog">Slow Log feature</a>.</p>
        
        <p>Additionally, you can use your favorite per-process monitoring program
        (top, htop, prstat, etc ...) to quickly check the CPU consumption of the
        main Redis process. If it is high while the traffic is not, it is usually
        a sign that slow commands are used.</p>
        
        <h2>Latency generated by fork</h2>
        
        <p>In order to generate the RDB file in background, or to rewrite the Append Only File
        if AOF persistence is enabled, Redis has to fork background processes.
        The fork operation (running in the main thread) can induce latency by itself.</p>
        
        <p>Forking is an expensive operation on most Unix-like systems, since it involves
        copying a good number of objects linked to the process. This is especially
        true for the page table associated to the virtual memory mechanism.</p>
        
        <p>For instance on a Linux/AMD64 system, the memory is divided in 4 KB pages.
        To convert virtual addresses to physical addresses, each process stores
        a page table (actually represented as a tree) containing at least a pointer
        per page of the address space of the process. So a large 24 GB Redis instance
        requires a page table of 24 GB / 4 KB * 8 = 48 MB.</p>
        
        <p>When a background save is performed, this instance will have to be forked,
        which will involve allocating and copying 48 MB of memory. It takes time
        and CPU, especially on virtual machines where allocation and initialization
        of a large memory chunk can be expensive.</p>
        
        <h2>Fork time in different systems</h2>
        
        <p>Modern hardware is pretty fast to copy the page table, but Xen is not.
        The problem with Xen is not virtualization-specific, but Xen-specific. For instance
        using VMware or Virutal Box does not result into slow fork time.
        The following is a table that compares fork time for different Redis instance
        size. Data is obtained performing a BGSAVE and looking at the <code>latest_fork_usec</code> filed in the <a href="/commands/info">INFO</a> command output.</p>
        
        <ul>
        <li><strong>Linux beefy VM on VMware</strong> 6.0GB RSS forked in 77 milliseconds (12.8 milliseconds per GB).</li>
        <li><strong>Linux running on physical machine (Unknown HW)</strong> 6.1GB RSS forked in 80 milliseconds (13.1 milliseconds per GB)</li>
        <li><strong>Linux running on physical machine (Xeon @ 2.27Ghz)</strong> 6.9GB RSS forked into 62 millisecodns (9 milliseconds per GB).</li>
        <li><strong>Linux VM on 6sync (KVM)</strong> 360 MB RSS forked in 8.2 milliseconds (23.3 millisecond per GB).</li>
        <li><strong>Linux VM on EC2 (Xen)</strong> 6.1GB RSS forked in 1460 milliseconds (239.3 milliseconds per GB).</li>
        <li><strong>Linux VM on Linode (Xen)</strong> 0.9GBRSS forked into 382 millisecodns (424 milliseconds per GB).</li>
        </ul>
        
        <p>As you can see a VM running on Xen has a performance hit that is between one order to two orders of magnitude. We believe this is a severe problem with Xen and we hope it will be addressed ASAP.</p>
        
        <h2>Latency induced by swapping (operating system paging)</h2>
        
        <p>Linux (and many other modern operating systems) is able to relocate memory
        pages from the memory to the disk, and vice versa, in order to use the
        system memory efficiently.</p>
        
        <p>If a Redis page is moved by the kernel from the memory to the swap file, when
        the data stored in this memory page is used by Redis (for example accessing
        a key stored into this memory page) the kernel will stop the Redis process
        in order to move the page back into the main memory. This is a slow operation
        involving random I/Os (compared to accessing a page that is already in memory)
        and will result into anomalous latency experienced by Redis clients.</p>
        
        <p>The kernel relocates Redis memory pages on disk mainly because of three reasons:</p>
        
        <ul>
        <li>The system is under memory pressure since the running processes are demanding
        more physical memory than the amount that is available. The simplest instance of
        this problem is simply Redis using more memory than the one available.</li>
        <li>The Redis instance data set, or part of the data set, is mostly completely idle
        (never accessed by clients), so the kernel could swap idle memory pages on disk.
        This problem is very rare since even a moderately slow instance will touch all
        the memory pages often, forcing the kernel to retain all the pages in memory.</li>
        <li>Some processes are generating massive read or write I/Os on the system. Because
        files are generally cached, it tends to put pressure on the kernel to increase
        the filesystem cache, and therefore generate swapping activity. Please note it
        includes Redis RDB and/or AOF background threads which can produce large files.</li>
        </ul>
        
        <p>Fortunately Linux offers good tools to investigate the problem, so the simplest
        thing to do is when latency due to swapping is suspected is just to check if
        this is the case.</p>
        
        <p>The first thing to do is to checking the amount of Redis memory that is swapped
        on disk. In order to do so you need to obtain the Redis instance pid:</p>
        
        <pre><code>$ redis-cli info | grep process_id&#x000A;process_id:5454&#x000A;</code></pre>
        
        <p>Now enter the /proc file system directory for this process:</p>
        
        <pre><code>$ cd /proc/5454&#x000A;</code></pre>
        
        <p>Here you&#39;ll find a file called <strong>smaps</strong> that describes the memory layout of
        the Redis process (assuming you are using Linux 2.6.16 or newer).
        This file contains very detailed information about our process memory maps,
        and one field called <strong>Swap</strong> is exactly what we are looking for. However
        there is not just a single swap field since the smaps file contains the
        different memory maps of our Redis process (The memory layout of a process
        is more complex than a simple linear array of pages).</p>
        
        <p>Since we are interested in all the memory swapped by our process the first thing
        to do is to grep for the Swap field across all the file:</p>
        
        <pre><code>$ cat smaps | grep &#39;Swap:&#39;&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                 12 kB&#x000A;Swap:                156 kB&#x000A;Swap:                  8 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  4 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  4 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  4 kB&#x000A;Swap:                  4 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;Swap:                  0 kB&#x000A;</code></pre>
        
        <p>If everything is 0 kb, or if there are sporadic 4k entries, everything is
        perfectly normal. Actually in our example instance (the one of a real web
        site running Redis and serving hundreds of users every second) there are a
        few entries that show more swapped pages. To investigate if this is a serious
        problem or not we change our command in order to also print the size of the
        memory map:</p>
        
        <pre><code>$ cat smaps | egrep &#39;^(Swap|Size)&#39;&#x000A;Size:                316 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  4 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  8 kB&#x000A;Swap:                  0 kB&#x000A;Size:                 40 kB&#x000A;Swap:                  0 kB&#x000A;Size:                132 kB&#x000A;Swap:                  0 kB&#x000A;Size:             720896 kB&#x000A;Swap:                 12 kB&#x000A;Size:               4096 kB&#x000A;Swap:                156 kB&#x000A;Size:               4096 kB&#x000A;Swap:                  8 kB&#x000A;Size:               4096 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  4 kB&#x000A;Swap:                  0 kB&#x000A;Size:               1272 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  8 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  4 kB&#x000A;Swap:                  0 kB&#x000A;Size:                 16 kB&#x000A;Swap:                  0 kB&#x000A;Size:                 84 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  4 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  4 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  8 kB&#x000A;Swap:                  4 kB&#x000A;Size:                  8 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  4 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  4 kB&#x000A;Swap:                  4 kB&#x000A;Size:                144 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  4 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  4 kB&#x000A;Swap:                  4 kB&#x000A;Size:                 12 kB&#x000A;Swap:                  4 kB&#x000A;Size:                108 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  4 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  4 kB&#x000A;Swap:                  0 kB&#x000A;Size:                272 kB&#x000A;Swap:                  0 kB&#x000A;Size:                  4 kB&#x000A;Swap:                  0 kB&#x000A;</code></pre>
        
        <p>As you can see from the output, there is a map of 720896 kB
        (with just 12 kB swapped) and 156 kb more swapped in another map:
        basically a very small amount of our memory is swapped so this is not
        going to create any problem at all.</p>
        
        <p>If instead a non trivial amount of the process memory is swapped on disk your
        latency problems are likely related to swapping. If this is the case with your
        Redis instance you can further verify it using the <strong>vmstat</strong> command:</p>
        
        <pre><code>$ vmstat 1&#x000A;procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----&#x000A; r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa&#x000A; 0  0   3980 697932 147180 1406456    0    0     2     2    2    0  4  4 91  0&#x000A; 0  0   3980 697428 147180 1406580    0    0     0     0 19088 16104  9  6 84  0&#x000A; 0  0   3980 697296 147180 1406616    0    0     0    28 18936 16193  7  6 87  0&#x000A; 0  0   3980 697048 147180 1406640    0    0     0     0 18613 15987  6  6 88  0&#x000A; 2  0   3980 696924 147180 1406656    0    0     0     0 18744 16299  6  5 88  0&#x000A; 0  0   3980 697048 147180 1406688    0    0     0     4 18520 15974  6  6 88  0&#x000A;</code></pre>
        
        <p><sup>C</sup></p>
        
        <p>The interesting part of the output for our needs are the two columns <strong>si</strong>
        and <strong>so</strong>, that counts the amount of memory swapped from/to the swap file. If
        you see non zero counts in those two columns then there is swapping activity
        in your system.</p>
        
        <p>Finally, the <strong>iostat</strong> command can be used to check the global I/O activity of
        the system.</p>
        
        <pre><code>$ iostat -xk 1&#x000A;avg-cpu:  %user   %nice %system %iowait  %steal   %idle&#x000A;          13.55    0.04    2.92    0.53    0.00   82.95&#x000A;&#x000A;Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util&#x000A;sda               0.77     0.00    0.01    0.00     0.40     0.00    73.65     0.00    3.62   2.58   0.00&#x000A;sdb               1.27     4.75    0.82    3.54    38.00    32.32    32.19     0.11   24.80   4.24   1.85&#x000A;</code></pre>
        
        <p>If your latency problem is due to Redis memory being swapped on disk you need
        to lower the memory pressure in your system, either adding more RAM if Redis
        is using more memory than the available, or avoiding running other memory
        hungry processes in the same system.</p>
        
        <h2>Latency due to AOF and disk I/O</h2>
        
        <p>Another source of latency is due to the Append Only File support on Redis.
        The AOF basically uses two system calls to accomplish its work. One is
        write(2) that is used in order to write data to the append only file, and
        the other one is fdatasync(2) that is used in order to flush the kernel
        file buffer on disk in order to ensure the durability level specified by
        the user.</p>
        
        <p>Both the write(2) and fdatasync(2) calls can be source of latency.
        For instance write(2) can block both when there is a system wide sync
        in progress, or when the output buffers are full and the kernel requires
        to flush on disk in order to accept new writes.</p>
        
        <p>The fdatasync(2) call is a worse source of latency as with many combinations
        of kernels and file systems used it can take from a few milliseconds to
        a few seconds to complete, especially in the case of some other process
        doing I/O. For this reason when possible Redis does the fdatasync(2) call
        in a different thread since Redis 2.4.</p>
        
        <p>We&#39;ll see how configuration can affect the amount and source of latency
        when using the AOF file.</p>
        
        <p>The AOF can be configured to perform an fsync on disk in three different
        ways using the <strong>appendfsync</strong> configuration option (this setting can be
        modified at runtime using the <strong>CONFIG SET</strong> command).</p>
        
        <ul>
        <li><p>When appendfsync is set to the value of <strong>no</strong> Redis performs no fsync.
        In this configuration the only source of latency can be write(2).
        When this happens usually there is no solution since simply the disk can&#39;t
        cope with the speed at which Redis is receiving data, however this is
        uncommon if the disk is not seriously slowed down by other processes doing
        I/O.</p></li>
        <li><p>When appendfsync is set to the value of <strong>everysec</strong> Redis performs an
        fsync every second. It uses a different thread, and if the fsync is still
        in progress Redis uses a buffer to delay the write(2) call up to two seconds
        (since write would block on Linux if an fsync is in progress against the
        same file). However if the fsync is taking too long Redis will eventually
        perform the write(2) call even if the fsync is still in progress, and this
        can be a source of latency.</p></li>
        <li><p>When appendfsync is set to the value of <strong>always</strong> an fsync is performed
        at every write operation before replying back to the client with an OK code
        (actually Redis will try to cluster many commands executed at the same time
        into a single fsync). In this mode performances are very low in general and
        it is strongly recommended to use a fast disk and a file system implementation
        that can perform the fsync in short time.</p></li>
        </ul>
        
        <p>Most Redis users will use either the <strong>no</strong> or <strong>everysec</strong> setting for the
        appendfsync configuration directive. The suggestion for minimum latency is
        to avoid other processes doing I/O in the same system.
        Using an SSD disk can help as well, but usually even non SSD disks perform
        well with the append only file if the disk is spare as Redis writes
        to the append only file without performing any seek.</p>
        
        <p>If you want to investigate your latency issues related to the append only
        file you can use the strace command under Linux:</p>
        
        <pre><code>sudo strace -p $(pidof redis-server) -T -e trace=fdatasync&#x000A;</code></pre>
        
        <p>The above command will show all the fdatasync(2) system calls performed by
        Redis in the main thread. With the above command you&#39;ll not see the
        fdatasync system calls performed by the background thread when the
        appendfsync config option is set to <strong>everysec</strong>. In order to do so
        just add the -f switch to strace.</p>
        
        <p>If you wish you can also see both fdatasync and write system calls with the
        following command:</p>
        
        <pre><code>sudo strace -p $(pidof redis-server) -T -e trace=fdatasync,write&#x000A;</code></pre>
        
        <p>However since write(2) is also used in order to write data to the client
        sockets this will likely show too many things unrelated to disk I/O.
        Apparently there is no way to tell strace to just show slow system calls so
        I use the following command:</p>
        
        <pre><code>sudo strace -f -p $(pidof redis-server) -T -e trace=fdatasync,write 2&gt;&amp;1 | grep -v &#39;0.0&#39; | grep -v unfinished&#x000A;</code></pre>
        
        <h2>Latency generated by expires</h2>
        
        <p>Redis evict expired keys in two ways:</p>
        
        <ul>
        <li>One <em>lazy</em> way expires a key when it is requested by a command, but it is found to be already expired.</li>
        <li>One <em>active</em> way expires a few keys every 100 milliseconds.</li>
        </ul>
        
        <p>The active expiring is designed to be adaptive. An expire cycle is started every 100 milliseconds (10 times per second), and will do the following:</p>
        
        <ul>
        <li>Sample <code>REDIS_EXPIRELOOKUPS_PER_CRON</code> keys, evicting all the keys already expired.</li>
        <li>If the more than 25% of the keys were found expired, repeat.</li>
        </ul>
        
        <p>Given that <code>REDIS_EXPIRELOOKUPS_PER_CRON</code> is set to 10 by default, and the process is performed ten times per second, usually just 100 keys per second are actively expired. This is enough to clean the DB fast enough even when already expired keys are not accessed for a long time, so that the <em>lazy</em> algorithm does not help. At the same time expiring just 100 keys per second has no effects in the latency a Redis instance.</p>
        
        <p>However the algorithm is adaptive and will loop if it founds more than 25% of keys already expired in the set of sampled keys. But given that we run the algorithm ten times per second, this means that the unlucky event of more than 25% of the keys in our random sample are expiring at least <em>in the same second</em>.</p>
        
        <p>Basically this means that <strong>if the database contains has many many keys expiring in the same second, and this keys are at least 25% of the current population of keys with an expire set</strong>, Redis can block in order to reach back a percentage of keys already expired that is less than 25%.</p>
        
        <p>This approach is needed in order to avoid using too much memory for keys that area already expired, and usually is absolutely harmless since it&#39;s strange that a big number of keys are going to expire in the same exact second, but it is not impossible that the user used <a href="/commands/expireat">EXPIREAT</a> extensively with the same Unix time.</p>
        
        <p>In short: be aware that many keys expiring at the same moment can be a source of latency.</p>
        
        <h2>Redis software watchdog</h2>
        
        <p>Redis 2.6 introduces the <em>Redis Software Watchdog</em> that is a debugging tool
        designed to track those latency problems that for one reason or the other
        esacped an analysis using normal tools.</p>
        
        <p>The software watchdog is an experimental feature. While it is designed to
        be used in production enviroments care should be taken to backup the database
        before proceeding as it could possibly have unexpected interactions with the
        normal execution of the Redis server.</p>
        
        <p>It is important to use it only as <em>last resort</em> when there is no way to track the issue by other means.</p>
        
        <p>This is how this feature works:</p>
        
        <ul>
        <li>The user enables the softare watchdog using te <code>CONFIG SET</code> command.</li>
        <li>Redis starts monitoring itself constantly.</li>
        <li>If Redis detects that the server is blocked into some operation that is not returning fast enough, and that may be the source of the latency issue, a low level report about where the server is blocked is dumped on the log file.</li>
        <li>The user contacts the developers writing a message in the Redis Google Group, including the watchdog report in the message.</li>
        </ul>
        
        <p>Note that this feature can not be enabled using the redis.conf file, because it is designed to be enabled only in already running instances and only for debugging purposes.</p>
        
        <p>To enable the feature just use the following:</p>
        
        <pre><code>CONFIG SET watchdog-period 500&#x000A;</code></pre>
        
        <p>The period is specified in milliseconds. In the above example I specified to log latency issues only if the server detects a delay of 500 milliseconds or greater. The minimum configurable period is 200 milliseconds.</p>
        
        <p>When you are done with the software watchdog you can turn it off setting the <code>watchdog-period</code> parameter to 0. <strong>Important:</strong> remember to do this because keeping the instance with the watchdog turned on for a longer time than needed is generally not a good idea.</p>
        
        <p>The following is an example of what you&#39;ll see printed in the log file once the software watchdog detects a delay longer than the configured one:</p>
        
        <pre><code>[8547 | signal handler] (1333114359)&#x000A;--- WATCHDOG TIMER EXPIRED ---&#x000A;/lib/libc.so.6(nanosleep+0x2d) [0x7f16b5c2d39d]&#x000A;/lib/libpthread.so.0(+0xf8f0) [0x7f16b5f158f0]&#x000A;/lib/libc.so.6(nanosleep+0x2d) [0x7f16b5c2d39d]&#x000A;/lib/libc.so.6(usleep+0x34) [0x7f16b5c62844]&#x000A;./redis-server(debugCommand+0x3e1) [0x43ab41]&#x000A;./redis-server(call+0x5d) [0x415a9d]&#x000A;./redis-server(processCommand+0x375) [0x415fc5]&#x000A;./redis-server(processInputBuffer+0x4f) [0x4203cf]&#x000A;./redis-server(readQueryFromClient+0xa0) [0x4204e0]&#x000A;./redis-server(aeProcessEvents+0x128) [0x411b48]&#x000A;./redis-server(aeMain+0x2b) [0x411dbb]&#x000A;./redis-server(main+0x2b6) [0x418556]&#x000A;/lib/libc.so.6(__libc_start_main+0xfd) [0x7f16b5ba1c4d]&#x000A;./redis-server() [0x411099]&#x000A;------&#x000A;</code></pre>
        
        <p>Note: in the example the <strong>DEBUG SLEEP</strong> command was used in order to block the server. The stack trace is different if the server blocks in a different context.</p>
        
        <p>If you happen to collect multiple watchdog stack traces you are encouraged to send everything to the Redis Google Group: the more traces we obtain, the simpler it will be to understand what the problem with your instance is.</p>
        
        <h2>APPENDIX A: Experimenting with huge pages</h2>
        
        <p>Latency introduced by fork can be mitigated using huge pages at the cost of a bigger memory usage during persistence. The following appeindex describe in details this feature as implemented in the Linux kernel.</p>
        
        <p>Some CPUs can use different page size though. AMD and Intel CPUs can support
        2 MB page size if needed. These pages are nicknamed <em>huge pages</em>. Some
        operating systems can optimize page size in real time, transparently
        aggregating small pages into huge pages on the fly.</p>
        
        <p>On Linux, explicit huge pages management has been introduced in 2.6.16, and
        implicit transparent huge pages are available starting in 2.6.38. If you
        run recent Linux distributions (for example RH 6 or derivatives), transparent
        huge pages can be activated, and you can use a vanilla Redis version with them.</p>
        
        <p>This is the preferred way to experiment/use with huge pages on Linux.</p>
        
        <p>Now, if you run older distributions (RH 5, SLES 10-11, or derivatives), and
        not afraid of a few hacks, Redis requires to be patched in order to support
        huge pages.</p>
        
        <p>The first step would be to read <a href="http://lwn.net/Articles/374424/">Mel Gorman&#39;s primer on huge pages</a></p>
        
        <p>There are currently two ways to patch Redis to support huge pages.</p>
        
        <ul>
        <li><p>For Redis 2.4, the embedded jemalloc allocator must be patched.
        <a href="https://gist.github.com/1171054">patch</a> by Pieter Noordhuis.
        Note this patch relies on the anonymous mmap huge page support,
        only available starting 2.6.32, so this method cannot be used for older
        distributions (RH 5, SLES 10, and derivatives).</p></li>
        <li><p>For Redis 2.2, or 2.4 with the libc allocator, Redis makefile
        must be altered to link Redis with
        <a href="http://libhugetlbfs.sourceforge.net/">the libhugetlbfs library</a>.
        It is a straightforward <a href="https://gist.github.com/1240452">change</a></p></li>
        </ul>
        
        <p>Then, the system must be configured to support huge pages.</p>
        
        <p>The following command allocates and makes N huge pages available:</p>
        
        <pre><code>$ sudo sysctl -w vm.nr_hugepages=&lt;N&gt;&#x000A;</code></pre>
        
        <p>The following command mounts the huge page filesystem:</p>
        
        <pre><code>$ sudo mount -t hugetlbfs none /mnt/hugetlbfs&#x000A;</code></pre>
        
        <p>In all cases, once Redis is running with huge pages (transparent or
        not), the following benefits are expected:</p>
        
        <ul>
        <li>The latency due to the fork operations is dramatically reduced.
        This is mostly useful for very large instances, and especially
        on a VM.</li>
        <li>Redis is faster due to the fact the translation look-aside buffer
        (TLB) of the CPU is more efficient to cache page table entries
        (i.e. the hit ratio is better). Do not expect miracle, it is only
        a few percent gain at most.</li>
        <li>Redis memory cannot be swapped out anymore, which is interesting
        to avoid outstanding latencies due to virtual memory.</li>
        </ul>
        
        <p>Unfortunately, and on top of the extra operational complexity,
        there is also a significant drawback of running Redis with
        huge pages. The COW mechanism granularity is the page. With
        2 MB pages, the probability a page is modified during a background
        save operation is 512 times higher than with 4 KB pages. The actual
        memory required for a background save therefore increases a lot,
        especially if the write traffic is truly random, with poor locality.
        With huge pages, using twice the memory while saving is not anymore
        a theoretical incident. It really happens.</p>
        
        <p>The result of a complete benchmark can be found
        <a href="https://gist.github.com/1272254">here</a>.</p>
      </article>
    </div>
    <script src='/js/foot.js'></script>
  </body>
</html>
